Collision-avoidance system for autonomous-capable vehicles

ABSTRACT

A collision-avoidance system for use with an autonomous-capable vehicle can continuously receive image frames captured of the roadway to determine drivable space in a forward direction of the vehicle. The system can determine, for each image frame, whether individual regions of the image frame depict drivable space. The system can do so using machine-learned image recognition algorithms such as convolutional neural networks generated using extensive training data. Using such techniques, the system can label regions of the image frames as corresponding to drivable space or non-drivable space. By analyzing the labeled image frames, the system can determine whether the vehicle is likely to impact a region of non-drivable space. And, in response to such a determination, the system can generate control signals that override other control systems or human operator input to control the brakes, the steering, or other sub-systems of the vehicle to avoid the collision.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/987,460 titled “COLLISION-AVOIDANCE SYSTEM FOR AUTONOMOUS-CAPABLEVEHICLE,” and filed May 23, 2018, which, is a continuation of U.S.patent application Ser. No. 15/631,990, titled “COLLISION-AVOIDANCESYSTEM FOR AUTONOMOUS-CAPABLE VEHICLE,” and filed Jun. 23, 2017.Applicant claims priority to and the benefit of each of suchapplications and incorporate all such applications herein by referencein its entirety.

BACKGROUND

Autonomous vehicles refer to vehicles which replace human drivers withsensors and computer-implemented intelligence, sensors and otherautomation technology. Under existing technology, autonomous vehiclescan readily handle driving with other vehicles on roadways such ashighways. However, certain environments and weather conditions canadversely affect the performance of certain sensors and the autonomousdriving system and can pose challenges to autonomous vehicles.

BRIEF DESCRIPTION OF THE DRAWINGS

The disclosure herein is illustrated by way of example, and not by wayof limitation, in the figures of the accompanying drawings in which likereference numerals refer to similar elements, and in which:

FIG. 1 is a block diagram illustrating an example collision-avoidancesystem as a part of an autonomous-capable vehicle, in accordance withexamples described herein;

FIG. 2 illustrates an example of an autonomous-capable vehicle thatincludes an implementation of a collision-avoidance system, inaccordance with examples described herein;

FIG. 3 is a block diagram illustrating an example training dataprocessing system receiving and processing training image frames andother training data, in accordance with examples described herein;

FIG. 4 is a flow chart describing an example method of operating anexemplary collision-avoidance system within an autonomous-capablevehicle, in accordance with examples described herein;

FIG. 5 is a flow chart describing an example method of operating anexemplary training data processing system, in accordance with examplesdescribed herein;

FIG. 6 illustrates an exemplary image frame received and processed by acollision-avoidance system, in accordance with examples describedherein; and

FIG. 7 is a block diagram that illustrates a computer system upon whichexamples described herein may be implemented.

DETAILED DESCRIPTION

As described herein, an autonomous-capable vehicle (“vehicle” forsimplicity) can be operated in autonomous, semi-autonomous, or manualmodes. In the autonomous or semi-autonomous modes of operation, acontrol system of the vehicle can operate the vehicle without or withlimited human operator input. For example, the control system can applya throttle to accelerate the vehicle, can steer the vehicle, can engagebrakes of the vehicle, and can operate auxiliary functions the vehicle(e.g., activate hazard lights). In these modes of operation, the controlsystem receives sensor output from an array of sensors to generatevehicle control signals to operate the vehicle. The sensor array caninclude a Light Detection And Ranging (LIDAR) sensor, one or morecameras, a radar sensor, a sonar sensor, etc. The sensors in the sensorarray enable the control system to detect obstacles, vehicles,pedestrians, cyclists, etc. on the road. The control system can, forexample, steer and decelerate the vehicle to prevent the vehicle fromcollisions based on sensor output from the sensor array.

However, because it generates control signals based on sensor outputfrom a plurality of sensors, the control system can be susceptible toerrors resulting from conflicting information conveyed by differentsensors. For instance, a camera in the sensor array may detect a cycliston the road but LIDAR sensor may have difficulty detecting the cyclist.In response to the outputs from the camera and the LIDAR sensor, thecontrol system can erroneously determine that there is no cyclist on theroad. In addition, certain sensors may have inconsistent or unreliableoutputs under certain conditions. For example, the LIDAR sensor can beinaccurate or inconsistent in adverse weather (e.g., rain, snow).Furthermore, the control system itself may fail due to hardware orsoftware failures. Accordingly, there is a need for a reliable andindependent collision-avoidance system that can control the vehicle toavoid imminent collisions.

Examples herein provide for an autonomous-capable vehicle that includesa collision-avoidance system that can operate independently of othersystems of the autonomous-capable vehicle (e.g., vehicle control systemfor controlling autonomous vehicle operations). The collision-avoidancesystem can engage the brakes of the vehicle based on analysis of imageframes captured by one or more forward-facing cameras of the vehicle. Inparticular, the collision-avoidance system can analyze the image framesto identify and label regions of drivable space in a forward directionof the vehicle. Regions in the image frames not identified as drivablespace can be treated as non-drivable space that the vehicle should notimpact. In addition, by analyzing the image frames, thecollision-avoidance system can compute distances from each of thenon-drivable space regions to the vehicle. In this manner, thecollision-avoidance system can determine a likelihood of the vehicleimpacting an identified region of non-drivable space based on thevehicle's speed and trajectory. Based on this determination, thecollision-avoidance system can generate a collision-avoidance signal toavoid a collision or minimize the impact of the collision independentlyof other vehicle systems (e.g., vehicle control system for autonomousoperations) or vehicle operator inputs (e.g., brake pedal input). Thecollision-avoidance signal can cause the vehicle to engage one or morebrakes, to make a steering and/or throttle adjustment, etc. to avoid thecollision or minimize the impact of the collision.

As used herein, the term “drivable space” can mean road surfaces (e.g.,paved road surfaces) that the vehicle can safely traverse withoutimpacting or colliding with other objects. As used herein, the term“non-drivable space” can mean regions that the vehicle cannot safelytraverse. Non-drivable space can include sidewalks, curbs, othervehicles, pedestrians, cyclists, walls, road medians, debris, etc. Incertain examples, non-paved road surfaces can be identified by thecollision-avoidance system as regions of non-drivable space.

According to embodiments, the collision-avoidance system can determineregions of drivable space in a forward direction of the vehicle byanalyzing the image frames to label appropriate pixels of the imageframes as being representative of drivable space. Thecollision-avoidance system can do so using a machine-learned modeltrained to identify regions of images (e.g., pixels or groups of pixels)that are representative of drivable space. In some examples, thecollision-avoidance system can determine a corresponding probabilitythat each pixel of an image frame is representative of drivable space.Based on the probability (e.g., the probability being over or under athreshold value), the collision-avoidance system can label the pixel asdrivable space. Regions of the image frame not labeled can be identifiedor treated as non-drivable space. The collision-avoidance system canfurther determine the distance and direction of the vehicle in relationto each region of non-drivable space.

According to embodiments, the machine-learned model can be generatedusing training data. The training data can include training imageframes. The training image frames can be captured by cameras can becaptured by cameras that are similar to the camera(s) on the vehicle.The pixels of each of the training image frames can be labeled asrepresenting drivable space and/or non-drivable space. The labeling ofthe training image frames can be performed algorithmically with theassistance of a human operator to correct for errors or to resolveconflicts in the analyses. Using the training data, the machine-learnedmodel can be generated. To improve the accuracy of the machine-learnedmodel in identifying drivable space represented in image frames, it isdesirable to have a large number of training image frames depicting avariety of road conditions that can be faced by the vehicle.

The generation of the machine-learned model can be performed by one ormore computers and data corresponding to generated machine-learnedmodels can be transmitted to the vehicle. In some examples, the vehiclecan receive the data over a wireless network connection (e.g., cellular,Wi-Fi, etc.). In addition to or as an alternative, the vehicle canreceive the data using a storage device (e.g., flash memory drive, SDcard, etc.) or via a physical connection (e.g., USB, wired connection,etc.). The vehicle can include storage devices (e.g., flash memory) tostore the data corresponding to the machine-learned model. The datastored in the storage devices can be updated periodically. Thus, theperformance of the collision-avoidance system can be improved over timeby updating the machine-learned models using updated training data.

In some examples, the machine-learned model can be a convolutionalneural network (CNN). The CNN can comprise a plurality of perceptrons(e.g., artificial neurons). The perceptrons can be arranged in layers.There can be an input layer, intermediate layers, and an output layer ofperceptrons in the CNN. The collision-avoidance system can input datacorresponding to the image frames to the CNN to analyze the image framesto identify regions of drivable space therein. In certainimplementations, the CNN can output, for a particular pixel, alikelihood that the particular pixel is representative of drivablespace. In other examples, the CNN can output a binary value for theparticular pixel, the binary value indicating whether the particularpixel represents drivable space.

In the examples described herein, the collision-avoidance system candetermine, based on the speed, trajectory, and acceleration of thevehicle, whether the vehicle is likely to impact a region ofnon-drivable space (e.g., an obstacle, a curb, a pedestrian). The speed,trajectory, and acceleration of the vehicle can be determined by asystem of the vehicle (e.g., the vehicle control system) or by thecollision-avoidance system. Upon determining that the vehicle is likelyto impact a region of non-drivable space, the collision-avoidance systemcan generate a collision-avoidance signal to cause the vehicle to avoidimpacting the region of non-drivable space. The collision-avoidancesignal can activate the brakes of the vehicle. The collision-avoidancesystem can do so independently of the vehicle control system and/oroperator inputs. In other words, the collision-avoidance signalgenerated by the collision-avoidance system can override control outputsof the vehicle control system and/or operator inputs to, for example,activate the brakes of the vehicle.

According to some examples, the collision-avoidance system can generatecollision-avoidance control signals to control other aspects of thevehicle upon determining that the vehicle is likely to impact a regionof non-drivable space. For example, the collision-avoidance system canalso generate a steering output to alter the trajectory of the vehicleto avoid impacting a region of non-drivable space. In addition, thecollision-avoidance system can generate a throttle output and anauxiliary output (e.g., to turn on hazard lights of the vehicle).

In certain implementations, the collision-avoidance system outputs canoverride vehicle control signals generated by the control system. Forexample, the control system may generate a control signal to increasethe throttle of the vehicle so as to accelerate the vehicle. In responseto determining that the vehicle is likely to impact non-drivable space,the collision-avoidance system can generate a collision-avoidance signalto override the control signal (e.g., to stop applying the throttle) andto engage the breaks of the vehicle. As such, the collision-avoidancesystem can control the vehicle, independently of the control system(and/or independently of human operator input), to avoid collisions.

Numerous examples are referenced herein in context of an autonomousvehicle or autonomous-capable vehicle, which refers to any vehicle whichis operated in a state of automation with respect to at least steering,propulsion and braking. When operated in an autonomous operationalstate, examples provide that a vehicle may ignore human or manualdriving input (e.g., accelerator or brake input, steering wheel input),such that the vehicle control system is the only “driver” of thevehicle. As described with various examples, the vehicle may ignorehuman or manual driving input when the vehicle is in an autonomousoperational state, with exception that the vehicle may respond todesignated human inputs such as inputs to switch the operational stateof the vehicle or to take a safety action (e.g., emergency brake). Asdescribed with various examples, the designated human inputs which cantransition the operational state of the vehicle may be with detectedwith respect to interfaces (e.g., mechanical actuators) that are notused for driving the vehicle. Thus, the interface for enabling a humandriver to switch the state of the autonomous vehicle may be separatefrom mechanisms such as steering wheels, accelerators (e.g., “gaspedals”), brakes, or gear shifters.

One or more aspects described herein provide that methods, techniquesand actions performed by a computing device are performedprogrammatically, or as a computer-implemented method. Programmaticallymeans through the use of code, or computer-executable instructions. Aprogrammatically performed step may or may not be automatic.

One or more aspects described herein may be implemented usingprogrammatic modules or components. A programmatic module or componentmay include a program, a subroutine, a portion of a program, a softwarecomponent, or a hardware component capable of performing one or morestated tasks or functions. In addition, a module or component can existon a hardware component independently of other modules or components.Alternatively, a module or component can be a shared element or processof other modules, programs or machines.

Furthermore, one or more aspects described herein may be implementedthrough the use of instructions that are executable by one or moreprocessors. These instructions may be carried on a computer-readablemedium. Machines shown or described with figures below provide examplesof processing resources and computer-readable mediums on whichinstructions for implementing some aspects can be carried and/orexecuted. In particular, the numerous machines shown in some examplesinclude processor(s) and various forms of memory for holding data andinstructions. Examples of computer-readable mediums include permanentmemory storage devices, such as hard drives on personal computers orservers. Other examples of computer storage mediums include portablestorage units, such as CD or DVD units, flash or solid state memory(such as carried on many cell phones and consumer electronic devices)and magnetic memory. Computers, terminals, network enabled devices(e.g., mobile devices such as cell phones) are examples of machines anddevices that utilize processors, memory, and instructions stored oncomputer-readable mediums. Additionally, aspects may be implemented inthe form of computer programs.

System Descriptions

FIG. 1 is a block diagram illustrating an example collision-avoidancesystem as a part of an autonomous-capable vehicle, in accordance withexamples described herein. In examples described, anautonomously-capable vehicle can operate without human action whichwould otherwise operate the vehicle. For example, in the context ofautomobiles, an autonomously driven vehicle can steer, accelerate,shift, brake and operate lighting components. Some variations alsorecognize that an autonomous-capable vehicle can be operated eitherautonomously or manually.

With reference to an example of FIG. 1, the autonomous-capable vehicle10 includes control systems to operate the vehicle 10 in autonomous,semi-autonomous, or manual modes of operation. In addition, the vehicle10 includes a collision-avoidance system 100 that operates independentlyof the control systems of the vehicle 10. In particular, examplesprovide for the vehicle 10 to include a vehicle interface system 90 toreceive primary control signals to control the vehicle 10. The vehicle10 can also have a plurality of modes of operation, including, forexample, a manual mode, a semi-autonomous mode, and an autonomous mode.During the manual and semi-autonomous modes of operation, the vehicleinterface system 90 can receive human operator input 83 as the primarycontrol signal to operate the vehicle 10. During autonomous andsemi-autonomous modes of operation of the vehicle 10, vehicle interfacesystem 90 can receive command input 85 outputted by AV control system180 to operate the vehicle 10. Examples provide that in one or moremodes of operation of the vehicle 10, the collision-avoidance system 100can operate to override the primary control signals (e.g., operatorinput 83, command input 85) to independently control the throttle,braking, steering, and auxiliary functions of the vehicle 10 to avoidcollisions with obstacles, pedestrians, curbs, other vehicles, etc. Thecollision-avoidance system 100 can continuously monitor the space andenvironment around the vehicle 10 and can distinguish regions of spacethat are drivable (e.g., paved road without any objects thereon) fromregions of space that are non-drivable (e.g., another vehicle, a fence,a curb, etc.). Based on the speed, trajectory, acceleration, and theprimary control signals, the collision-avoidance system 100 candetermine whether the vehicle 10 is likely, without intervention fromthe collision-avoidance system 100, to impact a region of spacedetermined to be non-drivable. In response to this determination, thecollision-avoidance system 100 can intervene to override the primarycontrol signals and independently control the throttle, steering,braking, and auxiliary functions of the vehicle 10 to avoid a collision.

The vehicle 10 can be equipped with a sensor array 130 that includesmultiple types of sensors such as a LIDAR sensor 135, a radar sensor140, a Sound Navigation and Ranging (SONAR) sensor 145, and a camera(s)150. The sensors in the sensor array 130 can combine to provide acomputerized perception of the space and environment surrounding thevehicle 10. In particular, the camera(s) 150 can include multiple setsof camera sensors (e.g., video camera, stereoscopic pairs of cameras,depth perception camera, infrared camera, wide-angle camera, etc.) tomonitor the environment surrounding the vehicle 10. In addition, theLIDAR 135, radar 140, and SONAR 145 sensors can detect objectssurrounding the vehicle 10 and their respective distances to the vehicle10. The output from the sensor array 130—sensor data 131—is received bya sensor interface 170. The sensor interface 170 can include a separateinterface device for each of the sensors in the sensor array 130. Forexample, the sensor interface 170 can include a LIDAR interface devicefor the LIDAR sensor 135 and a camera interface device for the camera(s)150. The sensor interface 170 can include hardware and/or other logicalcomponents that process the received sensor data 131. For instance, theLIDAR interface device receiving LIDAR data from the LIDAR sensor 135can include logic and processing resources (e.g., a field programmablegate array (“FPGA”), a digital signal processor (“DSP”), etc.) toconvert the received LIDAR data into a format that can be used by the AVcontrol system 180. As another example, the camera interface devicereceiving camera data (e.g., raw image data) from the camera(s) 150 canconvert the data into formats (e.g., JPEG or other compressed oruncompressed data formats) used by the AV control system 180. In certainimplementations, the sensor interface 170 can perform processing toimprove sensor data quality. For instance, the camera interface devicecan reduce noise, adjust exposure, perform gamma correction, adjustcolor, adjust focus, and perform other processing of the raw image data.The processed sensor data can be outputted as sensor data 171.

According to one implementation, the vehicle interface system 90 caninclude or control multiple vehicle interfaces, including a throttleinterface 92, a steering interface 94, a braking interface 96, andauxiliary interface 98. Each of the vehicle interfaces controls acorresponding subsystem of the vehicle 10. For example, the throttleinterface 92 controls the throttle of the engine of the vehicle 10 toaccelerate the vehicle 10. The steering interface 94 controls thesteering column of the vehicle 10 to steer the vehicle 10. The brakeinterface 96 controls one or more brakes of the vehicle 10 to deceleratethe vehicle 10. The auxiliary interface 98 controls one or moreauxiliary functions the vehicle 10 such as operating the hazard lights,etc. The vehicle interface system 90 can further include a controller 84to receive primary control signals (e.g., operator input 83, commandinput 85) that control the vehicle interfaces during normal operationsof the vehicle 10. The controller 84 can control the various vehicleinterfaces 92-98 based on the received primary control signals.

In certain implementations, the vehicle 10 can operate in a manual modeof operation, in which a human operator generates the primary controlsignal received by the vehicle interface system 90. In the manual mode,the vehicle interface system 90 receives human operator input 83 andgenerates control signal 119 to control the various vehicle subsysteminterfaces 92, 94, 96, 98. For example, the controller 84 of the vehicleinterface system 90 can receive a throttle input from an acceleratorpedal in the vehicle passenger cabin. Similarly, the controller 84 canreceive a steering input from a steering wheel and a brake input from abrake pedal. In some examples, the human operator input 83 can bereceived directly by the vehicle interfaces. For instance, the throttleinterface 92 can directly receive an operator throttle input and thebrake interface 96 can directly receive an operator brake input.

In some examples, the vehicle 10 can also operate in an autonomous modeand/or a semi-autonomous mode of operation. In these modes of operation,the AV control system 180 can generate command input 85, which isreceived by the vehicle interface system 90 as the primary controlsignal to operate the vehicle. The AV control system 180 can generatethe command input 85 based on the received processed sensor data 171.The command input 85 can include route information 87 and one or moreoperational parameters 89 which specify an operational state of thevehicle 10 (e.g., desired speed, acceleration, etc.). The controller 84generates control signals 119 based on the command input 85 for the oneor more vehicle interfaces 92, 94, 96, 98, to control throttle,steering, braking, and auxiliary vehicle functions.

In one implementation, the AV control system 180, in the semi-autonomousmode of operation, can control some of the subsystems of the vehicle 10.For instance, the AV control system 180 can generate command input 85 tocontrol the throttle subsystem 91 and the brake subsystem 95 of thevehicle 10. The steering subsystem 93 can be controlled by operatorinput 83 to the vehicle interface system 90 from a human operator of thevehicle 10. In other implementations, the AV control system 180, in thesemi-autonomous mode of operation, can control all operations of thevehicle 10, but can require periodic operator input 83 to indicate thatthe human operator is monitoring the progress of the vehicle 10 tooverride the command input 85 from the AV control system 180 ifnecessary.

According to embodiments, the vehicle 10 can include acollision-avoidance system 100 that operates independently from the AVcontrol system 180 and/or from the human operator of the vehicle 10. Thecollision-avoidance system 100 can override the command input 85generated by the AV control system 180 to prevent the vehicle 10 fromcolliding with objects in the vehicle's path. The collision-avoidancesystem 100 can also override operator input 83 from the human operatorof the vehicle 10.

In some implementations, the collision-avoidance system 100 can beimplemented on separate hardware and software from the AV control system180. In other words, the collision-avoidance system 100 can beimplemented on one or more dedicated processing resources (e.g., CPUs,GPUs, DSPs, FPGAs, ASICs, etc.) executing software instructionsdedicated for the collision-avoidance system 100. The softwareinstructions can also be stored memory dedicated for thecollision-avoidance system 100. In this manner, the collision-avoidancesystem 100 can be functional and provide safety redundancy in the eventof a hardware or software failure with the AV control system 180. Inother implementations, the collision-avoidance system 100 can beimplemented on hardware shared with the AV control system 180. Forinstance, the AV control system 180 and the collision-avoidance system100 can share sensors, processing resources, memory resources, and thelike. In this manner, the complexity and costs of implementing thecollision-avoidance system 100 on the vehicle 10 can be reduced. Inthese implementations, the processing resources of the vehicle 10 canexecute the instructions corresponding to the functionalities of thecollision-avoidance system 100 at a higher priority level thaninstructions corresponding to the AV control system 180 to ensure thatthe collision-avoidance system 100 can override control input 83generated by the AV control system 180.

In certain implementations, the camera(s) 150 and camera(s) 155 can bevideo cameras and/or stereoscopic camera sets that continually generateimage data (e.g., 30 image frames per second, 60 image frames persecond, 120 image frames per second, etc.). The cameras 150 and 155 canbe positioned to capture image frames in a forward direction of thevehicle 10.

According to examples, the collision-avoidance system 100 can include acamera interface 105 for receiving image frames 151 from the camera(s)of the sensor array 130. The camera interface 105 can also receive imageframes 156 from additional camera(s) 155. The additional camera(s) 155can be one or more cameras or camera sets dedicated for thecollision-avoidance system 100. The camera(s) 155 can include awide-angle, forward-facing camera. By receiving image frames 156 fromcamera(s) 155, the collision-avoidance system 100 can operate to preventthe vehicle 10 from colliding with objects in the vehicle's path evenwhen the camera(s) 150 of the sensor array 130 (or the sensor array 130itself) malfunctions. Furthermore, camera(s) 155 can be positioned in adifferent location (e.g., on the front bumper of the vehicle 10) ascompared with the camera(s) 150 to improve the field of vision (e.g.,reduce blind spots, increase angle of view, etc.) of thecollision-avoidance system 100. In some implementations, the camera(s)150 of the sensor array 130 provides sufficient reliability and field ofvision for the collision-avoidance system 100. In these implementations,the additional camera(s) 155 may not be needed, and, by combiningutilizing existing camera(s) 150 of the sensor array 130, the cost forimplementing the collision-avoidance system 100 can be reduced.

According to some examples, the camera interface 105 can includeimage-processing resources, such as provided with a field programmablegate array (“FPGA”) or a digital signal processor (“DSP”), whichreceives and/or processes image frames 151 and 156 from the camera(s)150 and 155. For example, the image-processing resources of the camerainterface 105 can perform image-processing operations to merge the imageframes 151 and 156. The collision-avoidance system 100 can perform imageanalysis on the merged image frame to generate the safety controloutputs. As an alternative, separate image analyses can be performed onthe image frames 151 and 156 and the results of the separate imageanalyses are reconciled to generate the safety control outputs. Inaddition, the image-processing resources of the camera interface 105 canalso perform image-processing operations to process raw image frames 151and 156 to, for example, adjust the exposure, contrast, and colorpalette of the raw image frames 151 and 156. The image-processingresources can also perform filtering, reduce noise, and perform croppingor shifting of the raw image frames 151 and 156. The image processingcan be performed in order to improve image quality and improveperformance and accuracy of the image analyses of thecollision-avoidance system 100 in identifying drivable space in theimages frames captured by the cameras 150 and 155. The camera interface105 can output processed image frames 106.

In one implementation, the image-processing resources of the camerainterface 105 can modify certain characteristics of the image frames 151and 156 to render the processed image frames 106 more similar to imageframes in the training data used to generate the machine-learned modelused by the collision-avoidance system 100. For example, the imageframes in the training data can exhibit certain characteristics withrespect to color palette, hue, exposure, and contrast. Theimage-processing resources of the camera interface 105 can performimage-processing operations to render the processed image frames 106 tobe similar in those characteristics as compared with the training dataimage frames. In particular, the image-processing resources of thecamera interface 105 can apply a hue or color filter to generate theprocessed image frames 106 to, for example, minimize effects caused bythe cameras 150 and 155 and/or by lighting conditions of the vehicle'senvironment. For instance, the image-processing resources of the camerainterface 105 can determine that the contrast of a particular imageframe 151 or 156 is higher than that of the training data image frames(e.g., average or median value). In response, the image-processingresources of the camera interface 105 can reduce the contrast togenerate processed image frame 106, such that the contrast of the imageframe 106 is closer to the average or median contrast of the trainingdata image frames. In this manner, accuracy and consistency with respectto identification of drivable space in the processed image frames 106 bythe collision-avoidance system 100 can be improved.

According to embodiments, the collision-avoidance system 100 includesimage analysis 110 that receives and analyzes the processed image frames106 from the camera interface 105. In various aspects, image analysis110 can identify drivable space depicted in the processed image frames106 by analyzing the processed image frames 106 and labeling appropriatepixels in the processed image frames 106 as depicting drivable space.For instance, pixels corresponding to a paved road on which the vehicle10 is traveling can be labeled by the image analysis 110 as drivablespace. In contrast, pixels corresponding to a pedestrian captured in theprocessed image frames 106 would not be labeled by the image analysis110 as drivable space. In some implementations, pixels corresponding toa pedestrian, other vehicles, objects, or obstacles captured in theimage frames 106 would be labeled as non-drivable space by the imageanalysis 110.

In some examples, the image analysis 110 can analyze the processed imageframes 106 using a machine-learned model 112. The machine-learned model112 can be generated based on training data and received by the vehicle10 via a communication interface (not shown in FIG. 1). Thecommunication interface can receive data corresponding to themachine-learned model 112 through a wireless connection (e.g., cellularlink, Wi-Fi link, etc.) or a physical interconnection (e.g., USBconnection). The data corresponding to the model 112 can be stored inone or more storage devices on the vehicle 10 that are accessible to thecollision-avoidance system 100. The model 112 can be updatedperiodically based on updated training data and received by the vehicle10. In this manner, the performance of the image analysis 110 and thecollision-avoidance system 100 can be improved over time as additionaltraining data is collected.

In various aspects, the training data can comprise training imageframes. The training image frames can depict real-world road conditionsand can be images captured by cameras similar to cameras 150 and 155.For example, cameras on a fleet of vehicles can capture the trainingimage frames as the vehicles operate under real-world conditions. Thetraining image frames can be analyzed and drivable space in the trainingimage frames can be identified. For instance, appropriate pixels of thetraining image frames can be labeled as representing drivable space. Theanalyses of the training image frames can be performed programmaticallywith human supervision, correction, and/or quality checking. In somecases, the training data can include sensor data. For example, eachtraining image frame can be associated with a set of LIDAR sensor data.The set of LIDAR sensor data can indicate LIDAR measurements taken atthe time the associated training image frame was captured. The LIDARdata can aid in the analyses of the training images. For instance, LIDARmeasurements can be used to identify objects depicted in the trainingimage frames that should not be labeled as drivable space.

According to embodiments, the model 112 can be a convolutional neuralnetwork (CNN) generated using the training data. The CNN model 112 cancomprise a plurality of layers, each of which can comprise a pluralityof artificial neurons. Using the CNN model 112, the image analysis 110can determine whether regions of the processed image frames 106 depictdrivable space. In some examples, the image analysis 110 can determine alikelihood (e.g., a confidence score) that a particular region of theprocessed image frame 106 depicts drivable space. In addition, differentimplementations of the image analysis 110 can perform the analysis atdifferent levels of resolution or detail. In some implementations, theimage analysis 110 can analyze the processed image frames 106 todetermine a likelihood of each individual pixel of the processed imageframes 106 depicting drivable space. In other implementations, the imageanalysis 110 can perform the analyses at lower resolution or detail todetermine likelihoods of groups of pixels (e.g., contiguous group of tenpixels) as depicting drivable space.

The image analysis 110 can generate labels 111 that identify thedrivable spaces depicted in the processed image frames 106. In someimplementations, the labels 111 can be output by the image analysis 110as matrix data structures. Each value in the matrix data structure of alabel 111 can correspond to a label of a pixel (or pixel group) of thecorresponding processed image frame 106. For instance, for a processedimage frame 106 having a resolution of 3840×2160 pixels, thecorresponding label 111 determined by the image analysis 110 can be amatrix data structure having 3840×2160 data values, each of the datavalues indicating the drivable space label determined for acorresponding pixel of the processed image frame 106. In someimplementations, the processed image frame 106 and the correspondinglabel 111 can be stored as one data structure (e.g., a multi-dimensionalarray or matrix).

The collision-avoidance system 100 can include a correction unit 115 toperform corrections in the labeling of the processed image frames 106.In doing so, the correction unit 115 can receive both the processedimage frames 106 and corresponding labels 111 determined by the imageanalysis 110. In one implementation, the corrections unit 115 canexamine multiple processed image frames 106 and their associated labels111 to remove outlier labels in the processed image frame 106. Forexample, one particular processed image frame 106 may have small region(e.g., a single pixel, a small group of pixels) not labeled asdrivable-space that is surrounded by another region(s) that are labeledas drivable-space. In such an instance, the correction unit 115 candetermine that the small region that is not labeled as drivable-space(or labeled as non-drivable space) by the image analysis 110 is anoutlier and can be ignored by collision-avoidance system 100. Suchoutliers can result from small (and non-threatening) objects on theroadway depicted in the processed image frame 106 being detected by theimage analysis 110. As another example, the correction unit 115 cancompare labels 111 generated for multiple processed image frames 106captured at around the same time to identify outliers that can beignored by the collision-avoidance system 100. In removing the outliersand other labeling errors, the correction unit 115 can generatecorrected labels 116.

According to embodiments, the collision-avoidance system 100 can includean impact analysis 120 that determines whether a forward-directioncollision involving vehicle 10 is about to occur. The impact analysis120 can do so based on information regarding the current trajectory,speed, and controls of the vehicle 10, including AV control data 86 andvehicle data 88, and the processed image frames 106 and/or the correctedlabels 116. Based on the processed image frames 106 and/or the correctedlabels 116, the impact analysis 120 can determine spatial parameters foreach region of non-drivable space in the processed image frames 106.Spatial parameters can include a position relative to the vehicle 10, adistance from the vehicle 10, a rate of travel (for moving objects), anda direction of travel (also for moving objects). In response todetermining that a collision is about to occur, the impact analysis 120can generate impact parameters for the controller 125. The controllercan generate collision-avoidance controls 124 to control varioussub-systems of the vehicle 10 to avoid colliding with the non-drivablespace or to minimize the impact of any such collision. Thecollision-avoidance controls 124 can include throttle control 126,steering control 127, brake control 128, and auxiliary control 129.

Autonomous-Capable Vehicle Description

FIG. 2 illustrates an example of an autonomous-capable vehicle thatincludes an implementation of a collision-avoidance system, inaccordance with examples described herein. In the below discussion ofFIG. 2, reference may be made to features and examples shown anddescribed with respect to FIG. 1. For instance, the vehicle 200illustrated in FIG. 2 can be vehicle 200 of FIG. 1. In addition,collision-avoidance system 250 in FIG. 2 can be the collision-avoidancesystem 100 illustrated in and described with respect to FIG. 1.

In the example illustrated in FIG. 2, the vehicle 200 includes anautonomous vehicle (AV) control system 210 that can control theoperation of the vehicle in semi-autonomous and/or fully-autonomousmodes of operation. In these modes of operation, the AV control system210 receives sensor output from sensors on the vehicle to generatecontrol signals to control one or more aspects of the vehicle'soperation. The AV control system 210 can be implemented using acombination of processing resources 211 and storage resources and cancomprise one or more general-purpose CPUs, dedicated GPUs, and memoryelements. The processing resources 211 can be centralized, distributedand/or include resources dedicated for specific resources. In operation,the processing resources 211 of the AV control system 210 can implementmodels, decision making algorithms, routing and trajectorydetermination, external communications and various other processes aspart of its normal operation.

Sensors on the vehicle can include a sensor array 215 positioned on topof the vehicle. The sensor array 215 can include a LIDAR sensor 216generating LIDAR measurements and one or more cameras 217 for capturingimage frames. The one or more cameras 217 can collectively generate agenerate images frames that depict a 360° perspective around the vehicle200. The one or more cameras 217 can include stereoscopic camera pairs,video cameras, infrared cameras, and/or other specialized imagecapturing devices. The sensor array 215 can further include otherranging or detection sensors such as radars, ultrasonic ranging sensors,etc. Additionally, peripheral sensors such as radars, ultrasonicsensors, sonars, or other types of detectors can be positioned insuitable locations about the vehicle 200 (e.g., front bumper, bumpercorners, side fenders, mirrors, rear bumper, etc.) to cover any blindspots of the sensor array 215. For example, multiple radar sensors 218may be distributed about a perimeter of the vehicle. A front-facingcamera 220 may also be positioned on the front bumper of the vehicle.Other cameras may be mounted to the exterior of the vehicle, or withinan interior of the windshield.

The vehicle 200 can include operational features (or devices) referredto as vehicle control subsystems 202, 204, 206, 208. The vehicle controlinterfaces may include throttle control subsystem 202, steering controlsubsystem 204, braking control subsystem 206, and auxiliary controlsubsystem 208. The subsystems can be controlled by control interfaces ofthe vehicle 200 (e.g., interfaces 92, 94, 96, and 98 of FIG. 1). Theoperational features shown are only examples, and more or feweroperational features of the vehicle 200 can be utilized with variationsto examples as described. In an example of FIG. 2, the operationalfacets are represented by interfaces which can be commanded or otherwisecontrolled by individual control systems of the vehicle 200.

According to some examples, the processing resources 211 can include oneor more processors, and/or programmatic and hardware interfaces whichprovide for control parameters, shown in FIG. 2 as controls 212, to becontinuously generated and signaled to the individual vehicle controlsubsystems 202, 204, 206, 208 of the vehicle 200 as the vehicle operatesautonomously or semi-autonomously. Thus, for example, the controls 212can be communicated from the processing resources 211 (and AV controlsystem 210) to the respective vehicle control interfaces 202, 204, 206,208. The processing resources 211 may determine one or more trajectoriesfor the vehicle, and then implement the trajectories via controls 212.The trajectories may define one or multiple possible trajectories of thevehicle for a given future interval. For example, the trajectories caninclude one or more primary trajectories of the vehicle, and/or afailsafe trajectory which the vehicle is to implement in the event theAV control system 210 has a cataclysmic failure.

In the example of FIG. 2, the vehicle 200 includes a collision-avoidancesystem 250 that independently controls one or more operational featuresof the vehicle 200. In doing so, the collision-avoidance system 250 cancontinuously receive image frames from cameras 217 and 220. Thecollision-avoidance system 250 can analyze the received image frames todetermine that the vehicle 200 is likely to impact a region ofnon-drivable space (e.g., another vehicle, a curb, a pedestrian, acyclist, etc.). In response to such a determination, thecollision-avoidance system 250 can generate collision-avoidance controlsto avoid or minimize such an impact. The collision-avoidance controlsgenerated by the collision-avoidance system can, independently of othercontrol systems of the vehicle (e.g., AV control system 210) or humanoperator input, control one or more operational features of the vehicle200. In one implementation, the collision-avoidance system 250 canindependently engage the brakes of the vehicle 200 via the brakingcontrol subsystem 206 upon determining that a collision is likely tooccur. In other implementations, the collision-avoidance system 250 canindependently control the steering of the vehicle 200 via the steeringcontrol subsystem 204 and engage the brakes of the vehicle 200 via thebraking control subsystem 206. In addition, the collision-avoidancesystem 250 can override the controls received by the throttle controlsubsystem 202 to disengage the throttle of the vehicle upon determiningthat a collision is imminent or likely to occur. Thus, human operatorinput and/or the AV control system 210 can be overridden by thecollision-avoidance system 250.

The functions performed by the collision-avoidance system 250 caninclude: continuously receiving image frames from the cameras 217, 220;analyzing the received image frames to identify drivable space depictedin each of the image frames; performing spatial analysis to determine arespective distance and direction of each region of non-drivable space(e.g., regions not identified as drivable space) from the vehicle 200;determining a respective probability the vehicle 200 will impact eachregion of non-drivable space based on the determined distance anddirection and based on vehicle data (e.g., speed, trajectory,acceleration, etc.); based on probabilities of impact, determiningcollision-avoidance controls 254 to mitigate, minimize, or avoid acollision. The identification of drivable space in each received imageframe can be performed by applying one or more machine-learned models.Using the machine-learned models, the collision-avoidance system 250 candetermine a probability that each pixel (or each group of pixels) ofeach image frame depicts drivable space. Pixels for which thecorresponding probabilities exceed a threshold can be labeled as“drivable space.” Conversely, pixels for which the correspondingprobabilities are below the threshold can be labeled as “non-drivablespace.”

The collision-avoidance system 250 can be implemented using acombination of processing resources 251 and memory resources 252. Theprocessing resources 251 can comprise one or more general-purpose CPUs,one or more dedicated processors (e.g., GPUs), and other localized ordistributed processing resources. The memory resources 252 can be usedto store data corresponding to one or more machine-learned models usedto analyze received image frames and identify drivable space andnon-drivable space depicted in each of the received image frames. Thememory resources 252 can further include cache memory to store receivedimage frames. For example, a received image frame can be analyzed andstored in the cache memory. In this manner, operations that requiremultiple image frames received over a period of time (e.g., correctionoperations) can be performed by the collision-avoidance system 250.

According to some implementations, the throttle 202, steering 204,braking 206, and auxiliary 208 can each be independently controlled byoutput of the collision-avoidance system 250 (e.g., collision-avoidancecontrols 254). The collision-avoidance controls 254 can override anyother input to the vehicle control subsystems 202, 204, 206, 208 such ascontrols generated by the AV control system 210 (e.g., AV controls 212)or human operator input. In this manner, when no imminent collision isdetected by the collision-avoidance system 250, the vehicle 200 iscontrolled by the human operator of the vehicle 200 or by the AV controlsystem 210. As soon as the collision-avoidance system 250 detects animminent collision and generates collision-avoidance controls 254, oneor more aspects of the vehicle's operations are controlled by theoutputs of the collision-avoidance system 250.

In one example, the collision-avoidance system 250 can be configured toreceive real-time data and analysis from the AV control system 210 todetermine whether to intervene and override the AV control system 210.For instance, the collision-avoidance system 250 can be configured toreceive data and information regarding objects and obstacles detected bythe AV control system 210. If the received data and information conflictwith analyses performed by the collision-avoidance system 250 (e.g.,collision-avoidance system 250 detects imminent collision with an objectnot identified by the AV control system 210), the collision-avoidancesystem 250 can determine to generate collision-avoidance controls 254 tooverride the output of the AV control system 210 (e.g., AV controls212). On the other hand, if the data and information received from theAV control system 210 is congruent with analyses performed by thecollision-avoidance system 250, the collision-avoidance system 250 candetermine to allow the AV control system 210 to control the vehicle 200to avoid the potential collision via AV controls 212. In addition, datareceived from the AV control system 210 can include data regarding thehealth of the AV control system 210. For instance, upon receiving dataindicating that the AV control system 210 is experience cataclysmicsoftware or hardware failures, the collision-avoidance system 250 canintervene to control one or more aspects of the operation of the vehicle200.

According to embodiments, the AV control system 210 can be implementedusing the processing resources 211 (e.g., shown located in the trunk ofthe vehicle 200), separate and independent of the collision-avoidancesystem 250. Likewise, the processing resources 251 of thecollision-avoidance system 250 can represent separate and independentresources from those of the AV control system 210. Thus, for example,the processing resources 211 and processing resources 251 can utilizeseparate (i) programming or logical platforms and architecture, (ii)input sensors (e.g., cameras), and (iii) communication busses forcommunicating with other components of the vehicle 200, including thosevehicle interfaces that are to be controlled by the respective system orunits. For instance, the camera 220 on the front bumper of the vehicle200 can be dedicated for the collision-avoidance system 250. In somevariations, the AV control system 210 and collision-avoidance system 250can include separate housings, as well as separate power busses orsources of power. In this manner, any hardware or power failuresaffecting the AV control system 210 can be contained and would notaffect the collision-avoidance system 250.

In other implementations, the AV control system 210 and thecollision-avoidance system 250 can share physical processing resourcesand memory resources. In other words, the processing resources 211 andprocessing resources 251 can be implemented on one or more sharedphysical processors (e.g., CPUs, GPUs, etc.). In such implementations,the software instructions and modules implementing the functionalitiesof the AV control system 210 and the collision-avoidance system 250 cannevertheless be separately maintained and executed by the one or moreshared physical processors. In one example, one or more shared physicalprocessors implement a first set of logical cores to execute softwareinstructions corresponding to the functionalities of the AV controlsystem 210 and a second set of logical cores to execute softwareinstructions corresponding to the functionalities of thecollision-avoidance system 250. In this manner, software and logicalfailures of the software implementing the AV control system 210 iscontained to the functionalities of the AV control system 210 itself anddo not affect the functionalities of the collision-avoidance system 250.In another example, software instructions corresponding to thecollision-avoidance system 250 can be executed by the one or more sharedphysical processors at a higher priority level than softwareinstructions corresponding to other systems of the vehicle 200 thatutilize the one or more shared physical processors, including the AVcontrol system 210. This can ensure that processing performed by thecollision-avoidance system 250 (e.g., analyzing image frames,identifying drivable and non-drivable space, spatial analyses,directional analyses) are performed in real-time to avoid or mitigatepotential collisions with detected objects in the path of travel of thevehicle 200.

In addition, the vehicle 200 can include one or more communicationinterfaces (not shown in FIG. 2) to receive data corresponding to theone or more machine-learned models used in the analyses performed by thecollision-avoidance system 250. The communication interfaces cancomprise cellular (e.g., 4G, 5G, LTE, etc.), Wi-Fi, and wired interfaces(e.g., wireline data connector). In this manner, the machine-machinelearned models can be periodically updated based on additional trainingdata.

Training Data Processing System Description

FIG. 3 is a block diagram illustrating an example training dataprocessing system receiving and processing training image frames andother training data, in accordance with examples described herein. Inthe below discussion of FIG. 3, reference may be made to features andexamples described with respect to FIGS. 1-2. For instance, anautonomous-capable vehicle 390 in communication with training dataprocessing system 300 of FIG. 3 may be the autonomous-capable vehicle 10of FIG. 1 or the autonomous-capable vehicle 200 of FIG. 2.

Referring to FIG. 3, the training data processing system 300 can receivetraining data 376 to generate or update a machine-learned model foridentifying drivable space in image frames for use incollision-avoidance systems in an autonomous-capable vehicle 390. Thetraining data processing system 300 can include a data collectioninterface 310 to receive training data 376, including training imageframes 377, associated sensor data 378, and associated location data379. The training data 376 can be received from a plurality of trainingdata collection vehicles 375 carrying various sensors and cameras tocontinuously capture training image frames 377, sensor data 378, andlocation data 379 as the training data collection vehicles 375 traverseroadways. The training data collection vehicles 375 can include sensors(e.g., cameras, LIDAR, radar, sonar, GPS, etc.) that are similar tothose equipped by the autonomous-capable vehicle 390 that receives themachine-learned model generated using the training data 376. Eachcaptured training image frame 377 can be associated with a set of sensordata 378 (e.g., LIDAR measurements, radar measurements, sonarmeasurements, etc.) and a set of location data 379 (e.g., GPS data). Thetraining data 376 can be received by the training data processing system300 from the training data collection vehicles 375 over a network 380,such as a cellular data network (e.g., LTE, HSPA). In otherimplementations, the training data 376 can also be received by thetraining data processing system 300 via a physical link (e.g., a serialbus connection) with the training data collection vehicles 375. Thetraining data 376 can also be transferred to the training dataprocessing system 300 by an operator or administrator via a data storagemedium (e.g., a flash memory card).

In some examples, the training data collection vehicles 375 are operatedby human drivers. In other examples, the training data collectionvehicles 375 can be autonomous-capable vehicles that can be operatedautonomously or semi-autonomously. Still further, autonomous-capablevehicle 390, which receives the machine-learned models generated by thetraining data processing system 300, can also transmit training data 376to the training data processing system 300. In effect, as more and moreautonomous-capable vehicles 390 are deployed, more training data 376 canbe made available to the training data processing system 300 to improvethe machine-learned models used in the vehicles' collision-avoidancesystems.

According to embodiments, the training data processing system 300includes a data collection interface 310 to receive and process thetraining data 376. The data collection interface 310 can include animage filter 311 to pre-process the received training image frames 377.The image filter 311 can perform a number of pre-processing operations,including altering the hue, color, contrast, and exposure of thetraining image frames 377 such that the resulting training image frames377 are generally uniform in such characteristics. The image filter 311can also detect and filter out training image frames 377 that areunsuitable for training the machine-learned model such as over-exposed,corrupted, or unusable image frames.

In the examples described herein, the training data processing system300 can include a supervised training sub-system 330, which receivestraining data 376, including filtered training image frames 377, andassociated sensor data 378 and location data 379. The supervisedtraining sub-system 330 generates supervised training results 331, usingwhich a model generator 360 of the training data processing system 300can generate or update machine-learned models for identifying drivablespace depicted in image frames for use in collision-avoidance systems ofautonomous-capable vehicles 390. The supervised training results 331 cancomprise image frames that are labeled for drivable space. For instance,each region (e.g., a pixel, a group of pixels, etc.) of each image framecan have an associated label identifying the corresponding region asdepicting drivable space or non-drivable space. To generate supervisedtraining results, the supervised training sub-system 330 can perform anumber of methods of analyses, each being efficient at identifying aparticular type of feature in the image frames. For instance, thesupervised training sub-system 330 can include an image analyzer (sensordata) 335 that labels the regions of training image frames 377 based onsensor data 378 associated with the image frames 377. Image analysisperformed by 335 can be particularly effective at identifyingnon-stationary features (e.g., pedestrians, other vehicles, etc.). Thesupervised training sub-system 330 can further include an image analyzer(location data) 340 that labels regions of training image frames 377based on location data 379 associated with the training image frames377. Image analysis performed by 340 can be particularly effective atidentifying stationary features (e.g., sidewalks, buildings, paved roadsurfaces) stored in a location or mapping database. The supervisedtraining sub-system can further include an image analysis (model) 345that identifies and labels the training image frame based on an existingmachine-learned model. In addition, the supervised training sub-system330 can further include a supervisory module 355 that receives anoperator input to label regions of training image frames 377 or tocorrect or supplement the labeling performed by analyzers 335 and 340.

In certain implementations, the image analyzer (sensor data) 335 canlabel training image frames 377 for drivable space and/or non-drivablespace based on sensor data 378 associated with the training image frames377. As part of this process the image analyzer (sensor data) 335 cangenerate a labeled image 336, which can be an image file with anassociated matrix of labels. For a particular training image frame 377captured by a training data collection vehicle 375, the image analyzer(sensor data) 335 analyzes an associated set of sensor data 378, whichcan include LIDAR, sonar, radar measurements taken by sensors of thetraining data collection vehicle 375 at the time the particular trainingimage frame 377 was captured. Using the sensor data 378, the imageanalyzer (sensor data) 335 can identify objects depicted in the trainingimage frame 377 and label regions of the training image frame 377corresponding to such objects as non-drivable space. Regions of thetraining image frame 377 not corresponding to objects detected using thesensor data 378 can be labeled as drivable space. The image analyzer(sensor data) 335 can be used to identify pedestrians, other vehicles,walls, or moving objects on the road depicted in the training imageframes 377. In certain examples, the image analyzer (sensor data) 335can perform image recognition to determine road surface types depictedin the training image frame 377 and perform labeling in accordance tothe determination. For instance, the image analyzer (sensor data) 335can distinguish between sidewalks (non-drivable space) and paved roads(drivable space).

According to embodiments, the image analyzer (location data) 340 canlabel training image frames 377 for drivable and/or non-drivable spacebased on location data 379 associated with the training image frames377. Similar to the image analyzer (sensor data) 335, the image analyzer(location data) 340 can generate a labeled image 341. For a particulartraining image frame captured by a training data collection vehicle 375,the image analyzer (location data) 340 analyzes an associated set oflocation data 379 (e.g., GPS data, GLONASS data, etc.) that indicates aprecise location of the training data collection vehicle 375 at the timethe particular training image frame 377 was captured. In addition, thelocation data 379 can further include heading information of thevehicle. The heading information can be determined using a compass ofthe training data collection vehicle 375 or based on location datacollected over time to determine a trajectory of the training datacollection vehicle 375. The image analyzer (location data) 340 can querya location or mapping database using the precise location and heading ofthe training image collection vehicle 375 and match nearby featuresidentifiable in the location or mapping database (e.g., paved roadsurfaces, sidewalks, buildings, other mapped features, etc.) to featuresdepicted in the training image frame 377. A feature in the location ormapping database can be matched to regions of the training image frames377 based on its location and direction relative to the training datacollection vehicle 375. As an example, using the precise location andheading of the training data collection vehicle 375, the image analyzer(location data) 340 can determine, by querying the location or mappingdatabase, that a region of the training image frame 377 depicts asidewalk or a wall. In response to such a determination, the imageanalyzer (location data) 345 can label the region of the training imageframe 377 as non-drivable space. As another example, using the preciselocation and heading of the training data collection vehicle 375, theimage analyzer (location data) 340 can determine that another region ofthe training image frame 377 depicts a paved road surface and can, inresponse, label that region of the training image frame 377 as drivablespace. The location or mapping database used by the image analyzer(location data) 340 can be managed by the training data processingsystem 300 or by a third-party mapping service.

According to embodiments, the results of various methods of imageanalyses and labeling can be reconciled and combined. As shown in FIG.3, the labeled images 336, 341, and 346 generated by the image analyzer(sensor data) 335, image analyzer (location data) 340, and imageanalyzer (model) 345, respectively, are reconciled and combined by labelreconciler 350. Since, as described above, each method of image analysiscan be effective at identifying a particular type of feature depicted inthe training image frames 377, the label reconciler 350 can combine thelabeled images 336 and 341 in a manner so as to maximize the efficacy ofthe combination. In the example depicted in FIG. 3, different methods ofimage analyses are performed by image analyzer (sensor data) 335 andimage analyzer (location data) 340. Analysis performed by image analyzer(location data) 340 can be effective at identifying stationary or mappedfeatures depicted in the training image frame 377 and analysis performedby image analyzer (sensor data) 335 can be effective at identifyingmoving features depicted in the training image frame 377. Thus, thelabel reconciler 350 can begin the reconciliation process with theresults of image analyzer (location data) 340—labeled image 341—thatidentifies stationary or mapped features depicted in the training imageframe 377. Then, the label reconciler 350 can overlay the results of theimage analyzer (sensor data) 335—labeled image 336—to arrive atpreliminary training results 351. In this manner, the label reconciler350 can begin with a depiction of the stationary or mapped features suchas paved road surfaces (drivable space) and can subsequently overlayidentified moving features such as another vehicle to arrive at thepreliminary training results 351. As an example, the label reconciler350 can begin with labeled image 341, which indicates that a certainregion of the training image frame 377 depicts drivable space (e.g.,paved road identified by image analyzer (location data) 340 usinglocation data 379). The label reconciler 350 can examine labeled image336 which indicates that the certain region depicts non-drivable space(e.g., another vehicle identified by image analyzer (sensor data) 335using sensor data 378). The resulting preliminary training results 351will indicate that the certain region corresponds to non-drivable spacebecause, even though it corresponds to a paved road, it is occupied byanother vehicle.

In certain implementations, the label reconciler 350, in combining theanalyses performed by image analyzer (sensor data) 335 and imageanalyzer (location data) 340, can identify issues corresponding toparticular regions of the training image frames 377. For instance, thelabel reconciler 350 may not be able to reconcile the analyses performedby image analyzer (sensor data) 335—labeled image 335—with the analysesperformed by image analyzer (location data) 340—labeled image 341 for aparticular portion of the training image frame 377. In response, thelabel reconciler 350 can identify the particular portion of the trainingimage frame 377 as having reconciliation issues. In turn, the particularportion of the training image frame 377 can be marked for supervisinginput 371 from an operator 370 to manually label the particular portionof the training image frame 377 as corresponding to drivable space ornon-drivable space.

In some examples, the supervised training sub-system 330 can furtherinclude a supervisory module 355 to receive supervising input 371 froman operator 370. The operator 370 can provide the supervising input 371to resolve issues in reconciling the labels to regions of the trainingimage frame 367 assigned by the image analyzer (sensor data) 335 and theimage analyzer (location data) 340. The operator 370 can further corrector supplement the programmatic analyses performed by the aforementionedfunctional blocks of the supervised training sub-system 330. Theoperator 370 can review the training image frame 367 and the associatedanalyses (e.g., preliminary training results 351) performed by the imageanalyzer (sensor data) 335 and the image analyzer (location data) 340.For instance, the operator 370 can view the training image frame 367 andthe preliminary training results 351, including the labels assigned tothe training image frame 367 indicating regions of drivable space andnon-drivable space as determined by the image analyzer (sensor data) 335and the image analyzer (location data) 340. The operator 370 can verifythe labels indicated in the preliminary training results 351 by matchingthem to the displayed training image frame, both being displayed on anoperator user interface (not shown in FIG. 3). If the operator 370determines that a certain region is mislabeled (e.g., a portion of theimage frame 367 corresponding to a vehicle being labeled as drivablespace), the operator 370 can provide supervising input 371 to correctthe mislabel. Based on the preliminary training results 351 and thesupervising input 371, the supervisory module 355 can generatesupervised training results 331 that incorporates the supervising input371 in correcting or supplementing the preliminary training results.

According to embodiments, the training data processing system 300 caninclude a model generator 360 to generate or update the machine-learnedmodel used to identify drivable space in captured image frames. Themodel generator 360 can generate or update the machine-learned model 361based on the supervised training results 331. In certain examples, themodel generator 360 can generate or update the machine-learned model 361as a convolution neural network. The training data processing system 300can further include a vehicle interface 320 for transmitting model data321 corresponding to the machine-learned model 361 to theautonomous-capable vehicle 390 over network 380. In other examples, themodel data 321 can also be transferred to the autonomous-capable vehicle390 via a direct data link (e.g., a bus connection) or a removablestorage device (e.g., a mass storage media device).

In certain implementations, virtual training image data 366 can also begenerated by a virtual training data generator 365 for use in generatingor updating the machine-learned model 361. The virtual training datagenerator 365 can be any computing system capable of generating virtualenvironments that correspond to simulated road conditions from aperspective similar to that of the cameras mounted on the training datacollection vehicles 375. Image frames of such virtual environments canbe captured by the virtual training data generator 365 as virtualtraining image frames 367. Other data regarding the virtual environmentscan be generated by the virtual training data generator 365 as virtualsensor data 368 and virtual location data 369. Examples can includecomputing systems for generating virtual reality or video game content.The data collection interface 310 and image filter 311 can process thevirtual training images frames 367 to be similar in characteristics(e.g., color, hue, contrast, exposure, etc.) to the training imageframes 377 captured by the cameras of the training data collectionvehicles 375.

Collision-Avoidance System Methodology

FIG. 4 is a flow chart describing an example method of operating anexemplary collision-avoidance system within an autonomous-capablevehicle, in accordance with examples described herein. In the belowdiscussion of FIG. 4, reference may be made to features and examplesshown and described with respect to FIGS. 1-3. For instance, the methodillustrated in FIG. 4 may be implemented and performed by the exemplarycollision-avoidance system 100 of FIG. 1 or the exemplarycollision-avoidance system 250 of FIG. 2.

Referring to FIG. 4, the exemplary collision-avoidance system canreceive an image frame from a camera mounted on the vehicle (410). Thecamera can be oriented in a direction of travel of the vehicle such thatthe captured images depict the roadway and other objects in front of thevehicle 10. The camera can continuously capture image frames fortransmission to the collision-avoidance system 100. For example, thecamera can capture image frames at a rate of 30 frames per second fortransmission to the collision-avoidance system. In turn, thecollision-avoidance system can process the received image frames insequence at the same rate of 30 frames per second.

In some implementations, the vehicle includes a single camera thatcaptures image frames for use by the collision-avoidance system. Inother implementations, the vehicle can include multiple cameras suchthat the collision-avoidance system 100 receives multiple image framesat the same time. The multiple cameras can be positioned at differentlocations on the vehicle to improve the field of vision or reduce blindspots of the collision-avoidance system. In addition, the multiplecameras (e.g., a stereoscopic camera pair) can be positioned to allowthe collision-avoidance system to sense depth. In implementations havingmultiple cameras, the collision-avoidance system can perform imageprocessing to combine or merge the images for analyses by thecollision-avoidance system. For instance, image frames captured by astereoscopic camera pair can be merged to create a three-dimensionalimage (e.g., an image with an associated depth map) for analyses by thecollision-avoidance system. As another example, image frames captured bymultiple cameras having complementary field of vision can be combined toproduce a combined image for analyses by the collision-avoidance system.

For each received image frame, the collision-avoidance system cananalyze the image frame to determine a respective probability that eachregion of the image frames depicts drivable space (415). In someimplementations, the collision-avoidance system can determine suchprobabilities on a pixel-by-pixel basis. Thus, for each individual pixelof each image frame received from the camera(s), the collision-avoidancesystem can determine a corresponding probability that the individualpixel represents drivable space (e.g., road surface that the vehicle cansafely traverse). In other examples, the network system can determinesuch probabilities for groups of pixels (e.g., a cluster of thirtypixels). The collision-avoidance system can perform step 415 byutilizing one or more machine-learned models for recognizing drivablespace in image frames. The machine-learned model can be generated usingextensive training data that includes a large number of training imagesthat have been previously analyzed for drivable space depicted therein.In at least one example, the machine-learned model utilized by thecollision-avoidance system is a convolution neural network.

Based on the determine probabilities, the collision-avoidance system canlabel individual regions of the received image frame as representingdrivable space or non-drivable space (420). For instance, if aparticular region's corresponding probability of representing drivablespace exceeds a certain threshold, the collision-avoidance system canlabel the particular region as representing drivable space. On the otherhand, if the particular region's corresponding probability ofrepresenting drivable space is below the threshold, thecollision-avoidance system can label the particular region asrepresenting non-drivable space.

After the received image frame is labeled for drivable space andnon-drivable space, the collision-avoidance system can correct thelabels for anomalies or outliers (425). The corrections can be performedon a single-image frame basis (426) wherein the image frame and theassociated labels are examined for anomalies or outliers. For example, asmall region (e.g., a single pixel, or a group of a few pixels) may belabeled as non-drivable space within a greater region that is otherwiseconsistently labeled as drivable space. The collision-avoidance systemcan determine the labeling of the small region as an outlier and alterthe corresponding labels to correct apparent mislabeling. In addition,corrections can be performed on a temporal basis by examining otherimage frames recorded at or around the same time as the received imageframe (427). The collision-avoidance system can compare the labelsassigned to multiple image frames recorded at or around the same time toidentify anomalies or outliers in labeling regions of the image frames'as drivable or non-drivable space. If the labeling between twocontemporaneously-recorded image frames are inconsistent, thecollision-avoidance system can determine to relabel some of the regionsof the image frames.

For each region of non-drivable space labeled in the received imageframe, the collision-avoidance system can determine correspondingspatial parameters (430). The spatial parameters can include a relativeposition to the autonomous-capable vehicle, a distance from theautonomous-capable vehicle, a rate of travel, and a direction of travel.Spatial parameters such as the relative position to and the distancefrom the autonomous-capable vehicle can be determined based on theposition of the non-drivable region within the image frame. Themachine-learned model can be further trained, based on training data andinput, to identify objects depicted in the received image frame andtheir distances to the autonomous-capable vehicle. Spatial parameterssuch as the rate of travel and the direction of travel can be calculatedby tracking the movement of the region of non-drivable space acrossmultiple image frames analyzed by the collision-avoidance system.

At step 435, the collision-avoidance system can determine a probabilitywhether the autonomous vehicle will impact a region of non-drivablespace. The determination can be based on the spatial parametersassociated with the non-drivable space as well as vehicle data such asthe vehicle's speed, direction, acceleration etc. If the correspondingprobability for each of the non-drivable spaces in the received imageframe is below a threshold value, the collision-avoidance system canmove on to the next image frame received from the camera (410).

On the other hand, if a probability of impact determined for at leastone region of non-drivable space of the received image frame exceeds thethreshold, the collision-avoidance system generate a collision-avoidancesignal to avoid the collision or minimize the impact of the collision(440). The collision-avoidance signal can override other vehicle controlsignals, such as those generated by the vehicle control system, whichcontrols the operation of the autonomous-capable vehicle insemi-autonomous or autonomous modes of operation. Thecollision-avoidance signal can cause the autonomous-capable vehicle toengage its brakes. In some examples, the collision-avoidance system canalso determine a steering control to steer the vehicle away from theregion of non-drivable space to avoid the impending collision. Thecollision-avoidance system can further receive data from the vehiclecontrol system of the autonomous-capable vehicle. The data can indicatewhether the vehicle control system has recognized an impending collisionmay occur. The collision-avoidance system can be configured to overridethe vehicle control signals generated by the vehicle control system onlywhen the data indicates that the vehicle control system has not detectedthat a collision is about to occur. After overriding the vehiclecontrols at step 440, the collision-avoidance system can continue toreceive image frames from the camera for analysis.

Training Data Processing System Methodology

FIG. 5 is a flow chart describing an example method of operating anexemplary training data processing system, in accordance with examplesdescribed herein. In the below discussion of FIG. 5, reference may bemade to features and examples shown and described with respect to FIG.3. For instance, the method illustrated in FIG. 3 may be implemented andperformed by the exemplary training data processing system 300 of FIG.3.

Referring to FIG. 3, the training data processing system receivestraining data (510). The training data can be collected by training datacollection vehicles (e.g., 375 of FIG. 3) as they traverse roadways. Thetraining data can comprise training image frames 511 and associatedsensor data 512. Each training image frame 511 can be associated with aset of corresponding sensor data 512 captured at the time the trainingimage frame 511 is recorded. The sensor data 512 can include LIDAR,sonar, and/or radar data. The training data can further includereal-time location data (e.g., GPS data).

According to embodiments, the training data processing system can applyappropriate image filters to the received training image frames (515).The training data processing system can do so to alter characteristicsof the received training image frames (e.g., color, hue, contrast,exposure, etc.). In doing so, the training data processing system canensure that the training image frames using which the machine-learnedmodel is generate are consistent in certain characteristics. This canimprove the quality and consistency of the machine-learned modelgenerated using the training image frames.

The training data processing system can analyze the training imageframes, associated sensor data, location data, and other data toidentify drivable and non-drivable space in the training image frames(520). The analysis can be performed using a variety of techniques. Forinstance, the training data processing system can identify regions ofdrivable space in a training image frame using a set of associate sensordata captured at the time the training image frame was recorded (521).Such analysis can be performed using, for example, LIDAR measurements.Based on the LIDAR measurements, the training data processing candetermine regions in the training image frame that correspond to a roadsurface. In response, the training data processing system can identifythose regions in the training image frame as drivable space. Inaddition, LIDAR measurements can further indicate regions in thetraining image frame corresponding to vehicles or pedestrians. Inresponse, the training data processing system can identify those regionsin the training image frame as non-drivable space.

Furthermore, the training data processing system can further analyzereal-time location data such as GPS data captured at the time thetraining image frame was recorded (522). Using the real-time locationdata, the training data processing system can determine regions of thetraining image frame that correspond to mapped features such asbuildings, sidewalks, road medians, etc. The training data processingsystem can do so by querying a location or mapping database. Based on adetermination that a particular region of the training image framecorresponds to a mapped feature such as a road median, the training dataprocessing system can identify and label the particular region of thetraining image frame as non-drivable space. In certain examples, thetraining image frame can also be analyzed by an existing iteration ofthe machine-learned model to identify drivable-space and/or non-drivablespace in the training image frame (523). The results of the analysis bythe existing model can be combined and reconciled with results fromother analyses to improve the existing iteration of the machine-learnedmodel.

According to embodiments, the training data processing system combinesand reconciles the various analyses to programmatically identify regionsof the training image frames as drivable or non-drivable space. Thecombined results can be presented to an administrator terminal forviewing by an administrator. In turn, the training data processingsystem can receive supervising input from the administrator inidentifying drivable or non-drivable space in the training image frames(525). The presentation on the administrator terminal can highlightparticular regions of the training image frames where the variousprogrammatic analyses cannot be reconciled or are in conflict with oneanother and thus require supervising input by the administrator. Forinstance, the presentation on the administrator terminal can highlight aparticular region of the training image frame using a different color ortheme as compared with the remainder of the training image frame. Theadministrator can, via the presentation displayed on the administratorterminal, provide supervising input identifying the particular region asa drivable or non-drivable space. In this manner, the administrator canprovide supervising input to reconcile, correct, or supplementprogrammatic analyses performed by the training data processing systemin analyzing training data for generating a machine-learned model toidentify drivable space in image frames captured by vehicle cameras.Accordingly, supervising input may be received for only a portion of thereceived training image frames. In other examples, the administrator canbe responsible for labeling the entire training image frame for drivablespace. Thus, the administrator can be responsible for “seeding” acollection of labeled training images with sufficient data points toenable the training data processing system to generate an initialmachine-learned model that can be subsequently improved with additionaltraining data.

At step 530, the machine-learned model can be generated or updated basedon the training data received at step 510. In certain embodiments, themachine-learned model is a convolution neural network. Still further,the machine-learned model can comprise a collection of classifiers. Thegenerated or updated machine-learned model can be transferred ortransmitted to autonomous-capable vehicles (535) for use with acollision-avoidance system deployed on the autonomous-capable vehicles.Data corresponding to the machine-learned model can be stored on astorage device on the autonomous-capable vehicle for access by thecollision-avoidance system.

Examplary Image Frame

FIG. 6 illustrates an exemplary image frame received and processed by acollision-avoidance system, in accordance with examples describedherein. In the below discussion of FIG. 6, reference may be made tofeatures and examples described with respect to FIG. 1. For instance,the exemplary image frame illustrated in FIG. 6 can be analyzed andprocessed by the collision-avoidance system 100 of FIG. 1.

Referring to FIG. 6, the illustrated exemplary image frame 600 can becaptured by one or more cameras of an autonomous-capable vehicle (e.g.,vehicle 10 of FIG. 1). The one or more cameras can be positioned tocapture the image frame 600 depicting the environment in front of theautonomous-capable vehicle. The collision-avoidance system can analyzethe image frame 600 to label various regions of the image frame 600 asrepresentative of drivable space or non-drivable space using amachine-learned model generated using training data. As can beappreciated, the image frame 600 is merely an illustration and variousfeature depicted in the image frame 600 are simplified for illustrativepurposes.

As illustrated, region 610 depicts a surface of a paved road. Thecollision-avoidance system can analyze the image frame 600 and label theregion 610 as drivable space using the machine-learned model. Region 620of image frame 600 depicts a pedestrian on walking on the paved road.The collision-avoidance system can analyze the image frame 600 using themachine-learned model and label region 620 as representing non-drivablespace. Similarly, for regions 630 and 635, which depict sidewalks, andregions 640 and 645, which depict walls, the collision-avoidance systemcan label those regions as non-drivable space.

In some examples, the collision-avoidance system can be configured torecognize regions of the image frame 600 as a region of the image frame600 above a horizon 655. The collision-avoidance system can ignore sucha region as being inconsequential in determining whether to generatecollision-avoidance signals. Furthermore, region 660 can correspond toan anomaly in the image frame 600. The anomaly can be caused by an errorof the image sensor of the camera or can be a lighting effect. In otherexamples, the anomaly can be a small object on the road that can besafely ignored by the collision-avoidance signal. As described above,the collision-avoidance system may initially label region 660 as anon-drivable space by applying the machine-learned model. However, usingerror-correction techniques, the anomaly can be detected and re-labeledas drivable space. In this manner, the collision-avoidance system willnot needlessly generate collision-avoidance signals to avoid impactingthe anomalous region.

In analyzing image frame 600, the collision-avoidance system candetermine that based on spatial parameters determined for non-drivablespace region 620 corresponding to the pedestrian and vehicle data (e.g.,speed, direction of travel, acceleration, etc.), impact with region 620is about to occur. In response, the collision-avoidance system cangenerate collision-avoidance signals to engage the brakes of the vehicleto avoid impacting the pedestrian.

Hardware Diagram

FIG. 7 is a block diagram that illustrates a computer system upon whichexamples described herein may be implemented. In the context of FIG. 1,the collision-avoidance system 100 may be implemented using a computersystem 700 such as described by FIG. 7. The collision-avoidance system100 may also be implemented using a combination of multiple computersystems as described in connection with FIG. 7.

In one implementation, the computer system 700 includes processingresources 710, a main memory 720, a read-only memory (ROM) 730, astorage device 740, and a communication interface 750. The computersystem 700 includes at least one processor 710 for processinginformation stored in the main memory 720, such as provided by a randomaccess memory (RAM) or other dynamic storage device, for storinginformation and instructions which are executable by the processor 710.The main memory 720 also may be used for storing temporary variables orother intermediate information during execution of instructions to beexecuted by the processor 710. The computer system 700 may also includethe ROM 730 or other static storage device for storing staticinformation and instructions for the processor 710. A storage device740, such as a magnetic disk or optical disk, is provided for storinginformation and instructions.

The communication interface 750 enables the computer system 700 tocommunicate with one or more networks 780 (e.g., cellular network)through use of the network link (wireless or wired). Using the networklink, the computer system 700 can communicate with one or more computingdevices and one or more servers. In accordance with examples, thecomputer system 700 receives model data 781 corresponding to amachine-learned model for identifying drivable and non-drivable spacesin image frames from over network 780 from one or more servers (e.g.,training data processing system 300 of FIG. 3). The model data 781 canbe received to update an existing machine-learned model 721 stored inthe main memory of the computer system 700. The executable instructionsstored in main memory 720 can further include instructions to performerror corrections on labeled image frames, instructions to determinespatial parameters corresponding to regions of image frames labeled asnon-drivable space, and instructions to determine whether, based onspatial parameters and vehicle data, impact with the region ofnon-drivable space is about to occur, and instructions to generatecollision-avoidance signals to control the vehicle and avoid theimminent collision. By way of example, the instructions and data storedin the memory 720 can be executed by the processor 710 to implement anexample collision-avoidance system 100 of FIG. 1. The processor 710 isconfigured with software and/or other logic to perform one or moreprocesses, steps and other functions described with implementations,such as described by FIGS. 1-6, and elsewhere in the presentapplication.

Examples described herein are related to the use of the computer system700 for implementing the techniques described herein. According to oneexample, those techniques are performed by the computer system 700 inresponse to the processor 710 executing one or more sequences of one ormore instructions contained in the main memory 720. Such instructionsmay be read into the main memory 720 from another machine-readablemedium, such as the storage device 740. Execution of the sequences ofinstructions contained in the main memory 720 causes the processor 710to perform the process steps described herein. In alternativeimplementations, hard-wired circuitry may be used in place of or incombination with software instructions to implement examples describedherein. Thus, the examples described are not limited to any specificcombination of hardware circuitry and software.

It is contemplated for examples described herein to extend to individualelements and concepts described herein, independently of other concepts,ideas or systems, as well as for examples to include combinations ofelements recited anywhere in this application. Although examples aredescribed in detail herein with reference to the accompanying drawings,it is to be understood that the concepts are not limited to thoseprecise examples. As such, many modifications and variations will beapparent to practitioners skilled in this art. Accordingly, it isintended that the scope of the concepts be defined by the followingclaims and their equivalents. Furthermore, it is contemplated that aparticular feature described either individually or as part of anexample can be combined with other individually described features, orparts of other examples, even if the other features and examples make nomentioned of the particular feature. Thus, the absence of describingcombinations should not preclude claiming rights to such combinations.

What is claimed is:
 1. A vehicle comprising: a sensor for capturingsensor data; a collision-avoidance system being configured to performoperations including: receiving sensor data from the sensor; determiningone or more first regions of the sensor data in which the vehicle cantraverse without colliding or impacting one or more objects within anenvironment of the vehicle and one or more second regions of the sensordata in which the vehicle cannot traverse without colliding or impactingthe one or more objects; determining that the vehicle is likely totravel into one of the second regions based at least in part on vehicledata indicative of a speed of the vehicle and a trajectory of thevehicle determined by a vehicle control system of the vehicle; and inresponse to determining that the vehicle is likely to travel into one ofthe second regions, generating a collision-avoidance signal thatoverrides the vehicle control system, the collision-avoidance signalbeing associated with an action that avoids a collision or minimizes theimpact of the collision of the vehicle associated with the secondregions of the sensor data.
 2. The vehicle of claim 1, wherein thevehicle control system is configured to (i) receive sensor input fromthe sensor, and (ii) generate, based on the sensor input, vehiclecontrol signals.
 3. The vehicle of claim 2, wherein thecollision-avoidance signal overrides at least one of the vehicle controlsignals generated by the vehicle control system.
 4. The vehicle of claim2, wherein the collision-avoidance system and the vehicle control systemutilize separate sets of hardware resources.
 5. The vehicle of claim 1,wherein determining the one or more first regions of the sensor data andthe one or more second regions of the sensor data comprises: determiningone or more probabilities associated with the one or more first regions.6. The vehicle of claim 5, wherein a probability of the one or moreprobabilities indicates the probability that the vehicle can traversethe respective first region without colliding or impacting the one ormore objects.
 7. The vehicle of claim 1, the collision-avoidance systembeing further configured to perform operations including: detecting theone or more objects within the environment of the vehicle.
 8. Thevehicle of claim 1, wherein determining that the vehicle is likely totravel into one of the second regions based at least in part on vehicledata indicative of the speed of the vehicle and the trajectory of thevehicle comprises: determining a spatial parameter for at least onesecond region, the spatial parameter indicating at least one of adistance between the vehicle and an object, a heading of the object, ora speed of the object.
 9. The vehicle of claim 1, wherein thecollision-avoidance signal causes the vehicle to activate one or morebrakes of the vehicle.
 10. A collision-avoidance system on a vehicle,the collision-avoidance system comprising circuitry configured to:receive sensor data from a sensor; determine one or more first regionsof the sensor data in which the vehicle can traverse without collidingor impacting one or more objects within an environment of the vehicleand one or more second regions of the sensor data in which the vehiclecannot traverse without colliding or impacting the one or more objects;determine that the vehicle is likely to travel into one of the secondregions based at least in part on vehicle data indicative of a speed ofthe vehicle and a trajectory of the vehicle determined by a vehiclecontrol system of the vehicle; and in response to determining that thevehicle is likely to travel into one of the second regions, generating acollision-avoidance signal that overrides the vehicle control system,the collision-avoidance signal being associated with an action thatavoids a collision or minimizes the impact of the collision of thevehicle associated with the second regions of the sensor data.
 11. Thecollision-avoidance system of claim 10, wherein the one or more firstregions of the sensor data and the one or more second regions of thesensor data are determined based at least in part on a machine-learnedmodel.
 12. The collision-avoidance system of claim 10, wherein thecollision-avoidance signal causes the vehicle to activate one or morebrakes of the vehicle.
 13. The collision-avoidance system of claim 10,wherein the collision-avoidance signal overrides at least one vehiclecontrol signal generated by the vehicle control system separate from thecollision-avoidance system.
 14. The collision-avoidance system of claim10, wherein the circuitry is further configured to: detect the one ormore objects within the environment of the vehicle.
 15. Thecollision-avoidance system of claim 14, wherein the one or more firstregions of the sensor data and the one or more second regions of thesensor data are determined based at least in part on the one or moreobjects detected within the environment of the vehicle.
 16. Thecollision-avoidance system of claim 10, wherein determining that thevehicle is likely to travel into one of the second regions based atleast in part on vehicle data indicative of the speed of the vehicle andthe trajectory of the vehicle comprises: determining a spatial parameterfor at least one second region, the spatial parameter indicating atleast one of a distance between the vehicle and an object, a heading ofthe object, or a speed of the object.
 17. The collision-avoidance systemof claim 10, wherein the sensor is a camera and wherein the sensor datais camera image data.
 18. A computer-implemented method of operating acollision-avoidance system on a vehicle, comprising: receiving sensordata from a sensor; determining one or more first regions of the sensordata in which the vehicle can traverse without colliding or impactingone or more objects within an environment of the vehicle and one or moresecond regions of the sensor data in which the vehicle cannot traversewithout colliding or impacting the one or more objects; determining thatthe vehicle is likely to travel into one of the second regions based atleast in part on vehicle data indicative of a speed of the vehicle and atrajectory of the vehicle determined by a vehicle control system of thevehicle; and in response to determining that the vehicle is likely totravel into one of the second regions, generating a collision-avoidancesignal that overrides the vehicle control system, thecollision-avoidance signal being associated with an action that avoids acollision or minimizes the impact of the collision of the vehicleassociated with the second regions of the sensor data.
 19. Thecomputer-implemented method of claim 18, further comprising: detectingone or more locations of the one or more objects within the environmentof the vehicle.
 20. The computer-implemented method of claim 19, whereindetermining the one or more first regions of the sensor data and the oneor more second regions of the sensor data comprises: determining the oneor more first regions of the sensor data and the one or more secondregions of the sensor data based at least in part on the one or morelocations of the one or more objects.